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BUILDING EFFECTIVE DATA 


==" STRATEGIES IN CAREER AND 
“mrs TECHNICAL EDUCATION 


ruatelge 


BUILDING KNOWLEDGE 
TO IMPROVE SOCIAL POLICY 


areer and technical education (CTE) programs (programs that 
teach students specific workplace skills aligned with the labor 
market) may track data for lots of different reasons: to comply 
with funding requirements, to manage their services and con- 
tinually improve, to measure their progress toward their goals, and to evaluate 
whether they are making a difference for students. When used intelligently, 
data can be powerful. Data can help a program refine its model, pinpoint suc- 


cesses, and communicate lessons with funders and stakeholders. 


Yet if data are so important, why don’t more programs have successful data strategies? 


Perhaps the reason is that there have been few guides available to help programs 
develop those strategies, even as CTE programs have received more attention and 
grown considerably in recent years. To fill that void, MDRC conducted a scan of leading 


CTE programs, starting with the projects affiliated with its Center for Effective CTE. 
MDEC research staff members and partners were interviewed about the challenges and 
opportunities data provide. MDRC also reached out to innovative leaders, consultants, 
and organizations in secondary, postsecondary, and workforce CTE — the three main 
areas of the field — to discuss their data strategies and the challenges they have faced. 


This brief presents highlights from those conversations. It further outlines four basic 
steps programs can take to strengthen their own CTE data-collection and measurement 
activities and develop robust — and manageable — data strategies. Because so few CTE 
programs have been subject to rigorous evaluation,’ many of the recommendations in 
this brief are based on the interviews outlined above, and supplemented by MDRC’s 
work with CTE programs to strengthen their design and implementation. 


1. Rachel Rosen, Mary Visher, and Katie Beal, “Career and Technical Education: Current Policy, 
Prominent Programs, and Evidence,” working paper (New York: MDRC, 2018). 


BACKGROUND 


The dearth of successful data strategies in 

CTE programs can be seen as related to the 
breadth of approaches that CTE programs 
employ — and the diversity of problems they 
are attempting to solve. As a result, all CTE 
programs looking to use data better can benefit 
from first answering two basic questions: What 
problem does the program address? And how 
does it do so? Answering these questions will 
help the program determine what data to collect 
to show whether it is meeting its goals. When 
MDRC works with programs, it encourages them 
to begin answering the questions by developing 
or refining a theory of change. A theory of 
change is a model that breaks down programs 
into essential components and mechanisms 
theorized to contribute to positive outcomes. 


Programs can use the theory-of-change 
framework to define what metrics will help 
them understand the activities and mediators 
(that is, the mechanisms of change) assumed 
to produce the desired outcomes (see Figure 1). 
They can then begin collecting data to answer 
questions about those outcomes. Not all outcomes 
will lend themselves to easy measurement, 

but the exercise of outlining outcomes can 
ensure that the data collection and analysis are 
focused on the most important components. 


Of course, once a program has determined what 
metrics will answer its questions it must collect 
the relevant data, and doing so can present 
challenges. One of these challenges is that a 
CTE program may wish to measure outcomes 
that require data from multiple institutions or 
systems. For example, a high school CTE program 
may feature a work-based learning component 
such as an internship and also offer classes that 
count for college credit. To measure outcomes, 
that program might need to collect data from 


secondary, postsecondary, and workforce data 
systems. Take another example: A workforce CTE 
program for adults may offer training toward a 
credential with the aim of increasing participants’ 
employment and earnings in the long term. To 
measure outcomes, that program would need to 
track the credentials participants earned from 
multiple certifying bodies, and would also need to 
obtain employment data on participants after they 
left the program, probably over months or years. 


Moreover, what defines many CTE programs 

is that they take an unconventional approach 

to education and to workforce preparation. 
Traditional data systems that focus on things 
like high school completion, postsecondary 
degree attainment, or program participation 

are not typically set up to provide the variety 

of data CTE programs need. Data on work- 
based learning participation, nontechnical skills 
acquisition, and the attainment of nondegree 
credentials such as certificates, certifications, or 
licenses can prove challenging to collect in an 
educational system not set up for tracking them.? 


To avert some of the challenges outlined in this 
paper, interviewees recommended hiring a data- 
strategy staff member early on. A person in this 
role can focus from the start on acquiring and 
analyzing data, devoting the time it takes to 
establish priority research questions, manage 
data collection and acquisition (by, for example, 


2. Nontechnical skills include those such as 
communication and teamwork. The terminology for 
nontechnical skills varies widely; related skills are 
described variously as employability skills, twenty- 
first-century skills, soft skills, job-readiness skills, 
career-ready competencies, noncognitive skills, 
essential skills, and workplace-readiness skills. 
Each term has its own distinguishing nuances. 
This brief uses the term “nontechnical skills” 
to encompass those skills that are essential to 
workplace experiences across industries. 
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developing data-sharing agreements or overseeing 
survey efforts), set up long-term data-collection 
systems and processes, and devise plans for 
analyzing the numbers. Of course, a data 
strategist alone cannot change an organization’s 
priorities; for that the organization needs the 
investment of both leaders and the front-line 

staff members who must enter the data. 


METRICS RELEVANT TO CTE: 
REALITIES AND WORK-AROUNDS 


Among the myriad activities and outcomes that 
programs track, a handful of components stand 
out as being both typical of CTE programs and 
challenging to monitor. Chief among them are 
work-based learning experiences, nontechnical 


skills acquisition, and credential attainment. This 
section summarizes challenges of, innovations 
in, and approaches to measuring each. 


Work-Based Learning Experiences 


While work-based learning experiences are 
integral to many CTE programs, their definitions 
and standards span a broad spectrum of activities. 
Work-based learning can include internships, 
apprenticeships, mentoring, long-term volunteer 
placements, job-shadowing opportunities, 

and more.® This variability creates questions 


3 Jessica Giffin, GeMar Neloms, Amanda Mitchell, 
and David Blumenthal, Work-Based Learning 
Definitions: Themes from States and National 
Organizations (Washington, DC: American 
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of both dosage (how much of a work-based 
learning experience a participant gets) and 
quality. For example, how do program operators 
compare the value, intensity, and goals of a visit 
to a local business with the value, intensity, 

and goals of a yearlong paid internship? 


While it may take rigorous research to determine 
the value of a work-based learning experience, a 
program can nonetheless establish a measurable 
target for itself and then develop a strategy to 
collect data that will reveal whether it has met that 
target. Here, developing or revisiting a program’s 
theory of change is an important first step. What 
level of exposure or type of experience is posited 
to produce the intended outcomes? The answer 
— and the related outcomes that the program 
deems important to track — will depend on the 
program and the context in which it operates. 
Once a program can clarify the purpose and 
function of a work-based learning experience, 

it will be in a better position to develop internal 
standards and consistent definitions, including 
standards for what “counts” as work-based 
learning and what aspects of an experience are 
most likely to yield the intended outcome. 


Though there is enormous variation in the 

field of work-based learning, the programs that 
responded to MDRC’s questions for this brief are 
using many preexisting pieces of software and 
other tools. For example, a matching platform 
like LaunchPath, which connects individuals with 


Institutes for Research, 2018). As an example of the 
extent of variation in definitions, 28 state education 
agencies have adopted formal definitions for work- 
based learning and 14 have adopted informal 
definitions; the remaining states have no definitions 
in place. Variations range from requirements that 
work-based learning experiences be paid, to 
references to employability or professional skills, 

to specifications that the work-based learning 
experiences connect to classroom learning. 


local experiences, may prove useful in answering 
questions of access such as: “Are participants 

of different races and genders being matched 
equitably?” To follow participation in work- 
based learning experiences, one CTE program 
has been using ImBlaze, an online portal and app 
that allows students and employers to connect 
for local internship opportunities, enter hours 
worked, and report information.* Regardless of 
how a program collects data on work-based 
learning experiences, it should prespecify its 
measurement targets and limit tracking and data 
collection to those that are most important. 


Nontechnical Competencies and 
Skills 


Increasingly, employers are seeking to hire 
people who can demonstrate they have learned 
nontechnical skills from work-based learning 
experiences. Measuring those skills can be 

just as thorny as tracking the experiences. The 
“nontechnical skills” employers may value range 
from broad designations like communication and 
problem solving to more discrete competencies 
like stress tolerance or persistence in striving 
toward goals. And although apparently 

similar nontechnical skills can be relevant to 
diverse contexts and situations, it is difficult to 
understand and measure an individual’s skill 
acquisition independent of context. For example, 
is peer communication in a welding shop a 
materially different skill from communicating 
with patients in a health care setting? 


First, programs must determine whether and 
why to track nontechnical skills. Here again, 


4  Inahandful of regions, schools, districts, and 
programs are also using ImBlaze to share 
information on work-based learning experiences in 
an effort to avoid counterproductive competition 
for the same opportunities. 
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developing and refining a concise theory of 
change can help programs decide whether 

to collect data on nontechnical skills, and to 
what end. Refining a theory of change and the 
rationale for collecting data will help programs 
determine which skills to track and how. For 
example, is the goal in collecting data to measure 
nontechnical skill acquisition for the program’s 
own processes of continual improvement? Or 

is it to help teachers steer instruction, or so 
that students can communicate their skills 
more effectively to potential employers? 


Relatedly, it is not always easy to figure out what 
nontechnical skills to track. There are many 
nontechnical skills programs could aim to 
improve, but they may initially do well to focus 
their data efforts on only a limited number. 


Once an organization identifies the skills aligned 
with an intervention’s intended outcomes, it 
must then determine how to measure those 
skills. The field has been grappling with this 
issue and some within it are attempting to 
standardize measures. The Regional Educational 
Laboratory of the Northeast and Islands has 
compiled a review of instruments that measure 
collaboration, perseverance, or self-regulated 
learning among secondary school students.° In 
the postsecondary space, the Education Design 
Lab’s BadgedToHire project, housed in three 
community colleges, is aimed at developing and 
expanding online microcredentialing assessments 
for hiring and employment. The New World 


5 Joshua Cox, Brandon Foster, and David Bamat, 
“A Review of Instruments for Measuring Social 
and Emotional Learning Skills Among Secondary 
School Students,” REL 2020-010 (Washington, 
DC: U.S. Department of Education, Institute for 
Education Sciences, National Center for Education 
Evaluation and Regional Assistance, Regional 


Educational Laboratory Northeast and Islands, 2019). 


All CTE programs looking to use 
Qata better can benefit from first 
answering two basic questions: 
What problem does the program 
address? And how does it do so? 


of Work has undertaken a similar effort in 

which students in high school and college can 
choose from 10 twenty-first-century skills and 
associated lessons, and online assessments of those 
skills are linked to digital badges that express 
their competencies. And separately, the ACT’s 
WorkKeys assessments bundle individual skills 
based on employers’ likely needs (for example, 
“graphic literacy” and “business writing”), thus 
supporting employer-to-program relationship 
building. These types of standardized assessments 
may be particularly useful for reporting to 
stakeholders on the skills students gain, or ifa 
CTE program’s mission makes it important to 
compare skills acquired across large groups. 


In some contexts, programs may prefer to review 
skills qualitatively. Many schools and districts have 
recently embraced performance-based learning 
through “capstones,” or performance-based 
assessments in which participants must undertake 
authentic tasks that mirror work responsibilities 

in those fields.® Such approaches seek to honor 

the complexity of learning. In addition, many 
platforms, such as MHA Labs, assess the growth of 
nontechnical skills through employer performance 
reviews, surveys, and structured sessions for 
comments and suggestions. However, these 
qualitative approaches may not be used as readily 


6 National Center for Innovation in Education, 
“Mapping Shifts in Teaching and Learning 
Supported by Performance Assessments” 
(website: www.leadingwithlearning.org/ 
performance-assessment-map, 2018). 
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to track program growth or make comparisons 
among students, employers, or organizations. 


Credential Attainment 


For programs that aim to help participants attain 
postsecondary credentials, it can be especially 
vexing to measure outcomes related to those 
credentials. For one thing, programs must 
determine which outcomes they can measure 
and are grounded in existing research. While 
an estimated 27 percent of adults in the United 
States have nondegree credentials (including 
certifications, licenses, and other postsecondary 
certificates that are not formal college degrees),’ 
not all credentials are created equal.® There 

is promising evidence that some nondegree 
credentials can lead to better wages, but those 
improvements are distributed unevenly across 
demographics, industries, and credentialing 
institutions.° In other words, just because a 


7 Stephanie Cronen, Meghan McQuiggan, Emily 
Isenberg, and Sarah Grady, Adult Training and 
Education: Results from the National Household 
Education Surveys Program of 2016 (Washington, 
DC: Institute for Education Sciences, National 
Center for Education Statistics, 2018). 


8 Sandy Baum, Higher Education Earnings 
Premium: Value, Variation, and Trends Connecting 
Credentials (Washington, DC: The Urban Institute, 
2014). 


9 Matthew D. Baird, Robert Bozick, and Melanie A. 
Zaber, “Beyond Traditional Academic Degrees: 
The Labor Market Returns to Occupational 
Credentials,” working paper (Santa Monica, CA: 
RAND Corporation, 2019); Mina Dadgar and 
Madeline Joy Weiss, “Labor Market Returns 
to Sub-Baccalaureate Credentials: How Much 
Does a Community College Degree or Certificate 
Pay?” CCRC Working Paper No. 45 (New York: 
Community College Research Center, 2012); David 
J. Deming, Noam Yuchtman, Amira Abulafi, Claudia 
Goldin, and Lawrence F. Katz, “The Value of 
Postsecondary Credentials in the Labor Market: An 


credential exists does not necessarily mean it 
has value. Still, in order to address questions 
of quality and value, programs first have to 
be able to keep track of what credentials their 
participants are earning and from where.'° 


On a systematic level, a few organizations are 
aiming to address the challenge of comparing the 
value of different credentials. Credential Engine 
is building a cloud-based registry to collect 

and connect information on all credentials in 
one publicly available catalog that would allow 
others (policymakers, states, credential issuers, 
researchers, etc.) to analyze the data and begin 

to make those comparisons related to quality." 
Connecting Credentials, a coalition of 121 
education and workforce cosponsors, has also 
sought to start a dialogue about transparency and 
the coordination of credentials. The coalition’s 
work includes a framework for making “apples- 
to-apples” comparisons of different credentials.‘ 


Despite these efforts, there are still areas where 
data about credentials are lacking. For example, no 
state has developed methods for tracking all types 
of nondegree credentials, nor is there a systematic 
federal data-collection effort for nondegree 
credentials.** In fact only about half of states 


Experimental Study” (American Economic Review 
106, 3: 778-806, 2016). 


10 Cronen, McQuiggan, Isenberg, and Grady (2018); 
ExcelinEd and Burning Glass Technologies, Report 
1: A National Landscape of High School Student 
Credential Attainment Compared to Workforce 
Demand (Tallahassee, FL: ExcelinEd, 2019). 


11. Credential Engine, “About Us — Credential Engine” 
(website: https://credentialengine.org/about, 2018). 


12 Connecting Credentials, “Connecting Credentials 
Framework” (website: http://connectingcredentials.org/ 
framework, 2016). 


13 Roberta Hyland, “Industry-Based Credentials Are 
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collect quantitative data on credential attainment, 
and most of those rely on data that schools or 
districts report themselves.'* As a consequence, if 
CTE programs want to track whether participants 
attain nondegree credentials, often they must 
reach out to those participants after they graduate, 
and doing so presents its own difficulties. 


While the field grapples with these systematic 
challenges, schools and programs can take 
concrete steps to keep tabs on credential 
attainment. CTE programs with partners 

or multiple locations can establish common 
data systems (using software like Salesforce, 
homegrown management information systems, 
or even systems of spreadsheets) to make it easier 
for those partners or locations to merge and 
aggregate data. Programs that rely on individual 
participants to report the credentials they attain 
can use survey platforms like SurveyMonkey, 
Qualtrics, or Google Forms to begin collecting 
that information quickly and inexpensively. 


BUILDING A DATA STRATEGY 


Both new CTE programs and mature 
programs looking to revisit their strategies 
need to establish clear plans for collecting and 
analyzing data. What follows is a series of steps 
a CTE program might take to that end. The 
recommendations in these steps are drawn 
from the interviews conducted during this 
scan and from MDRC’s experience supporting 
programs as they develop data processes. 


Part of the Education Pathway,” Clearinghouse 
Today (blog), National Student Clearinghouse 
(website: https://studentclearinghouse.org/ 
nscblog/industry-based-credentials-are-part-of- 
the-education-pathway, 2016); Jenna Leventoff, 
Measuring Non-Degree Credential Attainment 
(Washington, DC: National Skills Coalition, 2018). 


14 ExcelinEd and Burning Glass Technologies (2019). 


A handful of components stand 
out as being both typical of CTE 
programs and challenging to 
monitor. 


Steps for Effective Data 
Collection and Analysis 


Conduct a needs assessment and outline 

a theory of change. What issues in the 
community is the program trying to remedy? 
What problem does it aim to address? 
Delineating how its model is meant to bring 
about change can help a program not only 
identify the effects it is trying to achieve 

but also predict how program activities 
should lead to the anticipated outcomes. 
These predictions help programs understand 
which activities are important to measure. 


Define priority research questions and 
important outcomes. What does the 
program want to know or understand 

better in the short, medium, and long 

term? What is the intended effect of 
participating in the program? Answering 
questions like these means a CTE program 
can limit its data collection to top-priority 
areas. It can focus its time and effort on 
answering a few major questions, rather than 
collecting data on everything feasible and 
then struggling to focus its analysis. This 
recommendation may seem obvious, but 
when faced with expansive data sets, it can 
take a concerted effort to determine where to 
focus one’s often limited time and energy. 


With more clarity about research questions, 
a CTE program can identify which outcomes 
it must measure and what data it needs. 
When it is not feasible to collect data or 

the data will take a long time to obtain, 
interim and proxy outcomes can be a useful 
work-around. That is, are there shorter- 

term outcomes such as attendance that 
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can point toward longer-term payoffs like 
completion? Other interim outcomes to 
note will depend on the program’s goals, 
but examples may include perceptions and 
attitudes toward the program, academic 
improvement, or pay increases. 


In establishing an analysis plan, a program 

should return to its theory of change often, to 
ensure that the analysis is firmly grounded in 
the most important elements of the program. 


3 Set up data-collection processes. With 
important outcomes firmly established, a 
program can consider building a system 
to house data. While “data systems” can 
sound daunting, they can be as simple as 
spreadsheets or as creative as building on 
existing software. Interviewees reported that 
creating a new system can be easier (and more 
efficient in the long run) than retrofitting 
an older system to meet newer data needs. 


Programs can also reduce the amount of 
work staff members must put into collecting 
and tracking data by taking advantage of 
administrative data housed externally, in 
school districts, at community colleges, 

in statewide or national databases like 

the National Student Clearinghouse, or 
elsewhere.'* The accessibility of these 

data can vary widely, so programs may 
consider engaging a skilled technical- 
assistance provider or third party to navigate 
the sometimes confusing world of data 
collection and data-sharing agreements; 

the help may be particularly valuable 

when it comes to state systems, which 

may have detailed privacy regulations. 


15 Administrative data are those collected primarily for 
the management of programs and public services. 


4 §Iterate, adapt, and update. When in doubt, 
start somewhere. As one interviewee said, 
“You have to start to collect and report 
to be able to figure out where there are 
inconsistencies and gaps.” Be sure to share 
reports and analyses, even if they do contain 
inconsistencies or gaps, with staff members 
at all levels; seeing those reports can bring 
data entry to life and illuminate what is in 
it for teachers, employer partners, and staff 
members whose data you depend on — why it 
is worth their investment in time and energy. 
Ultimately, whatever dashboards, systems, 
or tools programs use to collect data, those 
systems should allow practitioners to access 
the data when they want. Practitioners can 
then, for example, track the hours students 
work week by week, or the percentage of them 
who attend a certain activity; such tracking 
can help programs respond to problems faster. 


POLICY IMPLICATIONS 


While CTE program administrators and staff 
members can take steps to strengthen their 

own data collection and analysis, some of the 
challenges outlined in this brief are likely to persist 
unless policies can be crafted to address them. 


One of the more persistent challenges centers on 
definitions and standardization. For example, 
credentialing bodies and institutions have not 
yet coordinated enough with one another for 
programs, employers, and institutions to make 
sense of the value of credentials, particularly 
credentials at the postsecondary level that fall 
short of associate’s degrees. Policy must address 
the needs of practitioners by establishing clear 
credentialing standards that allow for greater 
transparency and comparison. Similar problems 
exist when it comes to measuring work-based 
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learning. If CTE programs could standardize 
definitions in these areas, they could have 
more fruitful discussions among themselves 
— and discover where they need to change. 


Another policy challenge facing CTE programs 
relates to coordinating data: How are school 
districts, postsecondary institutions, and 
workforce programs communicating and sharing 
information and data? One promising step is 

that some organizations and coalitions have 
published tools and reports that attempt to bring 
into alignment the program-accountability 
indicators and teacher-quality provisions of the 
Workforce Innovation and Opportunity Act, 
Perkins V, and the Every Student Succeeds Act 
(federal legislation related to workforce training, 
career and technical education, and K-12 education 
respectively).'® While aligning these forms of data 
comes with challenges, it also presents compelling 
opportunities. Many CTE programs rely on 
partnerships and shared metrics. Establishing 
common metrics across partners leads to a richer 
understanding of what is happening in a program. 
Take the example of a hypothetical workforce 
CTE program: If that program only looks at 
employment and earnings outcomes, it may miss 
possible effects on postsecondary enrollment. 


16 Alliance for Excellent Education, the Center for 
American Progress, JFF, the Learning Policy 
Institute, the Linked Learning Alliance, NAF, 
and the National Center for the Improvement 
of Educational Assessment, Inc., Innovating for 
Equity and Excellence: Recommendations to 
States for Implementing the Strengthening Career 
and Technical Education for the 21st Century Act 
(Perkins V) (Washington, DC: Alliance for Excellent 
Education, 2019); Ellen Cushing, Susan Therriault, 
and David English, “Developing a College- and 
Career-Ready Workforce: An Analysis of ESSA, 
Perkins, and WIOA” (Washington, DC: American 
Institutes for Research, 2019). 


Just because a credential exists 
does not necessarily mean it has 
value. 


In the longer term, the field at large stands to 
gain a great deal from better metrics. Reliable 
measurement over a long span could allow the 
field to pinpoint the near-term measures that 
predict future workforce or college success. 


More broadly, better data can help ensure CTE 
doesn’t repeat the mistakes of its vocational- 
education past. Indeed, funders are increasingly 
paying attention to diversity, equity, and inclusion 
and asking for data on related outcomes. For 
example, Perkins V requires states to report data 
separated out according to several categories, 
including race and gender.’” CTE programs can 
get ahead of this trend by measuring equity-related 
outcomes, which can provide early opportunities 
to identify and address imbalances. Simply 
examining the outcomes of subgroups defined 

by race and ethnicity, gender, or socioeconomic 
status can reveal otherwise hidden inequities. 


While policymakers catch up to the new 
realities of CTE, programs can still take 

steps to make sure they are coordinating data 
strategies that work for them, by adopting 
new tracking tools, by revisiting their theories 
of change, or simply by getting started. 


17 Boris Granovskiy, Reauthorization of the Perkins 
Act in the 115th Congress: The Strengthening 
Career and Technical Education for the 
21st Century Act, R45446 (Washington, DC: 
Congressional Research Service, 2018). 
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